Detecting radio coverage problems

ABSTRACT

A method for detecting coverage problems is provided. The method includes receiving, at data processing hardware, from at least one user equipment (UE), observations. Each observation includes a signal measurement of a signal emitted from a base station and a corresponding location of the signal measurement. The method also includes generating, by the data processing hardware, a coverage map for the base station based on the received observations, the coverage map indicating a signal characteristic of the emitted signal about the base station. The method further includes determining, by the data processing hardware, an estimated characteristic of the base station by feeding the coverage map into a neural network configured to output the estimated characteristic of the base station.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 15/903,756,filed on Feb. 23, 2018, which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

This disclosure relates to detecting radio coverage problems.

BACKGROUND

Wireless communication networks (e.g., cellular networks) providecommunication content such as voice, video, packet data, messaging, andbroadcast for user equipment (UE), such as mobile devices and dataterminals. The communication network may include a number of basestations that can support communication for a number of user equipmentacross dispersed geographic regions.

In some configurations, wireless networks are rather large and mayemploy a large number of base stations. These larger networks may haveextensive site plans where telecommunication operators deploy many basestations (e.g., thousands of base stations). With respect to these basestations, site plans often dictate base station details, such as antennalocation, antenna feeder cables, antenna tilt angle, antenna azimuth,etc. Typically, these details and specifications for site plans havebeen preconfigured for network performance. Yet during networkoperation, it is not uncommon for user equipment to experience networkissues as a result of site plan deviation.

SUMMARY

One aspect of the disclosure provides a method for detecting radio orsignal coverage problems. The method includes receiving, at dataprocessing hardware, from at least one user equipment (UE)(e.g., anydevice capable of receiving signal emissions), observations. Eachobservation includes a radio signal measurement of a signal (e.g., aradio signal, WiFi signal, etc.) emitted from a base station and acorresponding location of the signal measurement. The method alsoincludes generating, by the data processing hardware, a coverage map forthe base station based on the received observations. The coverage mapindicates a signal characteristic of the emitted signal about the basestation. The method may further include generating, by the dataprocessing hardware, an observation map based on the coverage map andthe observations. The method includes determining, by the dataprocessing hardware, an estimated characteristic of the base station byfeeding the coverage map into a neural network configured to output theestimated characteristic of the base station. In some examples, themethod includes determining, by the data processing hardware, anestimated characteristic of the base station by feeding the coverage mapand the observation map into a neural network configured to output theestimated characteristic of the base station. In some implementations,the signal measurement includes a location uncertainty measurement.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, generating thecoverage map for the base station includes dividing a coverage areaabout the base station into pixels, each pixel corresponding to ageographical portion of the coverage area. Here, for each observation,the method may include identifying the pixel having the correspondinggeographical portion of the coverage area that contains the location ofthe signal measurement of the respective observation and associating theobservation with the identified pixel. For each pixel, the method mayinclude averaging the signal measurements of any observations associatedwith the respective pixel. For example, associating the observation withthe identified pixel may include placing the observation in a pixel binassociated with the identified pixel.

In some examples, the coverage map includes a grid having cells, andeach pixel corresponds to one of the cells. Generating the observationmap may also include, for each pixel of the coverage map, generating ametric that monotonically expresses a number of any observationsassociated with the respective pixel. When generating the coverage mapfor the base station, the method may include generating a terrain map ofa geographical area about the base station and feeding the terrain mapinto the neural network. The terrain map may describe at least one of aterrain altitude of the geographical area or a presence and/or height ofobjects extending above a ground surface of the geographical area.

In some configurations, when generating a metric that monotonicallyexpresses a number of any observations associated with the respectivepixel, the metric includes at least one of: a number of any observationsassociated with the respective pixel; a log of the number of anyobservations associated with the respective pixel; or a monotonicfunction of the number of any observations associated with therespective pixel. Additionally or alternatively, the metric may includedetermining a number of any observations associated with the respectivepixel, when the number is greater than zero, assigning a value of therespective pixel to one and when the number equals zero, assigning thevalue of the respective pixel to zero.

In some implementations, the method includes feeding side informationinto the neural network, the side information including at least one ofa frequency of operation of the base station, a height of an antenna ofthe base station, an antenna beam width, an antenna tilt angle, or apredetermined location of the base station. The characteristic of thebase station may include an estimated location of the base station, anestimated pointing direction of the base station, or an antenna azimuthof the base station. Optionally, the neural network may be configured tooutput a confidence indicator of the estimated characteristic of thebase station.

Optionally, the method includes generating, by the data processinghardware, a location uncertainty map based on location uncertaintymeasurements. The method may also include determining, by the dataprocessing hardware, the estimated characteristic of the base station byfeeding the coverage map and the location uncertainty map into theneural network configured to output the estimated characteristic of thebase station.

In some examples, generating the location uncertainty map for the basestation includes dividing a coverage area about the base station intopixels, each pixel corresponding to a geographical portion of thecoverage area. Here, for each observation, the method may includeidentifying the pixel having the corresponding geographical portion ofthe coverage area that contains the location of the location uncertaintymeasurement of the respective observation and associating theobservation with the identified pixel. For each pixel, the method mayinclude averaging the location uncertainty measurements of anyobservations associated with the respective pixel.

Another aspect of the disclosure provides a system for detecting signalcoverage problems. The system includes data processing hardware andmemory hardware in communication with the data processing hardware, thememory hardware storing instructions that when executed on the dataprocessing hardware cause the data processing hardware to performoperations. The operations includes receiving from at least one userequipment (UE), observations, each observation comprising a signalmeasurement of a signal emitted from a base station and a correspondinglocation of the signal measurement. The operations also includegenerating a coverage map for the base station based on the receivedobservations, the coverage map indicating a signal characteristic of theemitted signal about the base station and generating an observation mapbased on the coverage map and the observations. The operations mayfurther include generating an observation map based on the coverage mapand the observations. The operations also include determining anestimated characteristic of the base station by feeding the coverage mapinto a neural network configured to output the estimated characteristicof the base station. In some examples, the operations includedetermining an estimated characteristic of the base station by feedingthe coverage map and the observation map into a neural networkconfigured to output the estimated characteristic of the base station.In some implementations, the signal measurement includes a locationuncertainty measurement.

This aspect may include one or more of the following optional features.In some examples, the operation of generating the coverage map for thebase station includes dividing a coverage area about the base stationinto pixels, each pixel corresponding to a geographical portion of thecoverage area. For each observation, the operation may includeidentifying the pixel having the corresponding geographical portion ofthe coverage area that contains the location of the signal measurementof the respective observation and associating the observation with theidentified pixel. For each pixel, the operation may include averagingthe signal measurements of any observations associated with therespective pixel. For example, associating the observation with theidentified pixel may include placing the observation in a pixel binassociated with the identified pixel.

In some implementations, the coverage map includes a grid having cells,and each pixel corresponds to one of the cells. The operation ofgenerating the observation map may include, for each pixel of thecoverage map, generating a metric that monotonically expresses a numberof any observations associated with the respective pixel. The metric mayinclude at least one of a number of any observations associated with therespective pixel, a log of the number of any observations associatedwith the respective pixel, or a monotonic function of the number of anyobservations associated with the respective pixel. Optionally,generating the metric may include determining a number of anyobservations associated with the respective pixel, when the number isgreater than zero, assigning a value of the respective pixel to one andwhen the number equals zero, assigning the value of the respective pixelto zero.

In some configurations, the operation of generating the coverage map forthe base station includes generating a terrain map of a geographicalarea about the base station and feeding the terrain map into the neuralnetwork. The terrain map may describe at least one of a terrain altitudeof the geographical area or a presence and/or height of objectsextending above a ground surface of the geographical area. Optionally,the operations may include feeding side information into the neuralnetwork, the side information including at least one of a frequency ofoperation of the base station, a height of an antenna of the basestation, an antenna beam width, an antenna tilt angle, or apredetermined location of the base station. The estimated characteristicof the base station may include an estimated location of the basestation, an estimated pointing direction of the base station, or anantenna azimuth of the base station. The neural network may beconfigured to output a confidence indicator of the estimatedcharacteristic of the base station.

Optionally, the operation include generating a location uncertainty mapbased on location uncertainty measurements. The operations may alsoinclude determining the estimated characteristic of the base station byfeeding the coverage map and the location uncertainty map into theneural network configured to output the estimated characteristic of thebase station.

In some examples, generating the location uncertainty map for the basestation includes dividing a coverage area about the base station intopixels, each pixel corresponding to a geographical portion of thecoverage area. Here, for each observation, the operations may includeidentifying the pixel having the corresponding geographical portion ofthe coverage area that contains the location of the location uncertaintymeasurement of the respective observation and associating theobservation with the identified pixel. For each pixel, the system mayinclude averaging the location uncertainty measurements of anyobservations associated with the respective pixel.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspect features and advantages will be apparent from the description anddrawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of an example communication network.

FIG. 2A is a schematic view of an example coverage map.

FIGS. 2B and 2C are schematic views of example observation maps.

FIGS. 2D and 2E are schematic views of example terrain maps.

FIG. 3A is a schematic view of an example modeler with correspondinginputs and outputs.

FIGS. 3B and 3C are schematic views of example modelers with illustratedestimated outputs.

FIG. 4 is a schematic view of an example communication network with datastores.

FIG. 5 is a flow chart of an example method for detecting signalcoverage problems.

FIG. 6 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Like any project, telecommunication operators may have to modify siteplans during the deployment of base stations. For example, it generallyproves difficult to deploy a large number of base stations without somedegree of deviation from original site plans. Telecommunicationoperators may have to move antenna locations (e.g. due to sitedeployment terrain) or swap feeder cables (e.g., cable supply issues).In other examples, telecommunication operators simply make inadvertenterrors when deploying the infrastructure for networks. In the case oflarge networks, small errors may be amplified over a large site and/orsystem. Whether these deviations are minor or major, these deviationsmay affect later network performance during network operation. Issuescaused by these deviations may even go undetected for long periods oftime. Unfortunately, to verify the operation of each and everytransmitter (e.g., base station) is usually prohibitively expensive. Asa result, there is a need to detect signal coverage problems duringnetwork operation. With user equipment (UE) devices operating within ageographic coverage area of the base station, server devices receivinginformation from the UE devices may use predictive modeling to detectcharacteristics for a given base station. An advantage of this UEfeedback detection based system is that the network provider or networkmanager may virtually verify characteristics of network infrastructure(e.g. base stations) without a need to physically verify the networkinfrastructure. The concepts disclosed may also be applied to signalcoverage detection for WiFi access point and other signal emittingdevices.

FIG. 1 depicts an example communication network 100, which may be aLong-Term Evolution (LTE) network, a 5G network, and/or a multipleaccess network supporting numerous access technologies specified by the3^(rd) Generation Partnership Project (3GPP), such as the General PacketRadio Service (GPRS), the Global System for MobileCommunications/Enhanced Data Rates for GSM Evolution (GSM/EDGE), theUniversal Mobile Telecommunication System/High Speed Packet Access(UMTS/HSPA), LTE and LTE advanced network technologies. LTE is astandard for wireless communication of high-speed data packets betweenuser equipment 102, 102 a-c, such as mobile phones and data terminals,and base stations 104. LTE is based on the GSM/EDGE and UMTS/HSPAnetwork technologies. LTE is configured to increase the capacity andspeed of the telecommunication by using different radio interfaces inaddition to core network improvements. LTE supports scalable carrierbandwidths, from 14 MHz to 20 MHz and supports both frequency divisionduplexing (FDD) and time-division duplexing (TDD). In other examples,the communication network 100 is a WiFi network or other wireless signalnetwork. The user equipment 102 may be interchangeably referred to asuser equipment (UE) devices and mobile devices 102.

The UE devices 102, 102 a-c may communicate with an external network 30,such as a packet data network (PDN), through the communication network100 (or 5G/3G/2G network). In the example shown, the network 100includes a first portion, an Evolved Universal Terrestrial Radio AccessNetwork (e-UTRAN) portion 106, and a second portion, an Evolved PacketCore (EPC) portion 108. The first portion 106 includes an air interface110 (i.e., Evolved Universal Terrestrial Radio Access (e-UTRA)) of3GPP's LTE upgrade path for mobile networks, UE devices 102, and basestation 104. The LTE air interface 110 uses orthogonalfrequency-division multiple access (OFDMA) radio-access for the downlinkand Single-carrier FDMA (SC-FDMA) for the uplink. Accordingly, the firstportion 106 provides a radio access network (RAN) that supports radiocommunication of data packets and/or other surfaces from the externalnetwork to the UE devices 102 over the air interface 110 via one or morebase stations 104.

The EPC 108 provides a framework configured to converge voice and dataon the communication network 100. The EPC 108 unifies voice and data onan Internet Protocol (IP) service architecture and voice is treated asjust another IP application. The EPC 108 includes several key componentsthat include, without limitations, a Mobility Management Entity (MME)112, a Serving Gateway (SGW) 114, and a Packet Data Node Gateway (PGW)120. The PGW 120 may be referred to as a network gateway device 120, andwhen the network corresponds to a 3G network, the network gateway device120 includes a Gateway GPRS Support Node (GGSN) instead of the PGW 120.Optionally, when the network corresponds to a 5G or 5G+ network, thenetwork gateway device 120 may include a gateway node with a namingconvention as defined by the 5G and/or 5G+ network. The MME 112, the SGW114, and the PGW 120 may be standalone components, or at least two ofthe components may be integrated together. The EPC 108 communicates withthe UE devices 102 and the external network 30 to route data packetstherebetween.

The MME 112 is a key control-node for the communication network 100. TheMME 112 manages sessions and states and authenticates and tracks a UEdevice 102 across the network 100. For instance, the MME 112 may performvarious functions such as, but not limited to, control of signaling andsecurity for a Non Access Stratum (NAS), authentication and mobilitymanagement of UE devices 102, selection of gateways for UE devices 102,and bearer management functions. The SGW 114 performs various functionsrelated to IP data transfer for UE devices 102, such as data routing andforwarding, as well as mobility anchoring. The SGW 114 may performfunctions such as buffering, routing, and forwarding of data packets forUE devices 102. The SGW 114 and the MME 112 also communicate with oneanother over an S11 interface 113.

The PGW 120 (i.e., network gateway device) performs various functionssuch as, but not limited to, internet protocol (IP) address allocation,maintenance of data connectivity for UE devices 102, packet filteringfor UE devices 102, service level gating control and rate enforcement,dynamic host configuration protocol (DHCP) functions for clients andservers, and gateway general packet radio service (GGSN) functionality.The PGW 120 communicates with the SGW 114 over an S5 interface 115.

Each base station 104 has a coverage area 104 _(Area) that generallycorresponds to a geographic area where a UE device 102 may receive aradio signal S emitted from the base station 104. In someimplementations, the base station 104 includes an evolved Node B (alsoreferred as eNode B or eNB). An eNB 104 includes hardware that connectsto the air interface 110 (e.g., a mobile phone network) forcommunicating directly with the UE devices 102. For instance, the eNB104 may transmit downlink LTE/3G/5G signals (e.g., communications) tothe UE devices 102 and receive uplink LTE/3G/5G signals from the UEdevices 102 over the air interface 110. The eNBs 104 use an S1 interface111 for communicating with the EPC 108. The S1 interface 111 may includean S1-MME interface for communicating with the MME 112 and an S1-Uinterface for interfacing with the SGW 114. Accordingly, the S1interface 111 is associated with a backhaul link for communicating withthe EPC 108. In additional implementations, the base station 104 is awireless access point or other wireless signal emitter.

UE devices 102 may be any telecommunication device that is capable oftransmitting and/or receiving voice/data over the network 100. UEdevices 102 may include, but are not limited to, mobile computingdevices such as laptops, tablets, smart phones, and wearable computingdevices (e.g., headsets and/or watches). UE devices 102 may also includeother computing devices having other form factors, such as computingdevices included in desktop computers, vehicles, gaming devices,televisions, or other appliances (e.g., networked home automationdevices and home appliances).

In some implementations, data processing hardware 122 of the networkgateway device 120 (e.g., PGW or GGSN or a gateway node with anothernaming convention as defined by 5G and/or 3G+ networks) receives from atleast one UE device 102 (e.g., shown as UE 102 a-c) observations 130(i.e. observation data). The data processing hardware 122 may receivethe observations 130 based on interaction(s) the at least one UE device102 has with the network 100 within the coverage area 104 _(area) of thebase station 104. Each observation 130 includes a radio signalmeasurement 132 of the radio signal S emitted from the base station 104and a corresponding location 134 of the radio signal measurement 132.The location 134 refers to coordinates of the UE device 102 at a timethe observation 130 is transmitted. For example, the location 134 is aglobal longitude and/or latitude of the UE device 102. In otherexamples, the location 134 refers to a position relative to an object,such as the base station 104 emitting the radio signal S measure by anobservation 130. In some examples, the radio signal measurement 132 is ameasurement of signal strength. Some examples of measurements of signalstrength include received signal strength indicators (RSSI), referencesignal received power (RSRP), reference signal received quality (RSRQ),and/or timing advance. In some implementations, each observation 130corresponds to a particular base station 104. In other implementations,each observation 130 corresponds to more than one base station 104(e.g., adjacent base stations near the UE 102). When each observation130 corresponds to more than one base station 104, the observation 130may be parsed according to each observed base station 104. For example,the network gateway device 120 determines a particular base station 104corresponding to an observation 130 received from an UE device 102 andmay estimate characteristics of the particular base station 104 based onthe observation 130 or a collection of observations 130 from multiple UEdevices 102 within the coverage area 104 _(Area) corresponding to thebase station 104.

Referring to FIG. 1, the communication network 100 further includes amapper 200 and a modeler 300. The mapper 200 and/or the modeler 300 maybe implemented by the data processing hardware 122 of the networkgateway device 120. In some examples, the mapper 200 and/or the modeler300 are executed by data processing hardware corresponding to theexternal network 30. For example, the external network 30 may be adistributed system (e.g., a cloud environment) with its own dataprocessing hardware or shared data processing hardware (e.g., sharedwith the network gateway device 120). In yet other examples, the mapper200 and the modeler 300 are implemented on different data processinghardware in communication within the communication network 100.

Generally, the mapper 200 is configured to generate maps 210, 210 a-nfor a given base station 104 based on the received observations 130corresponding to that base station 104. Some examples of maps 210generated by the mapper 200 for a given base station 104 include acoverage map 210 a, an observation map 210 b, and a terrain map 210 c.Each of these maps 210 may include different types of map information212 relating to the received observations 130. In some examples, eachmap 210 represents the coverage area 104 _(area) for a base station 104,but varies with regard to map information 212 depicted within thatcoverage area 104 _(area). For instance, a map 210 is divided intogeographic portions 214 (i.e. geographic subsections) sometimes referredto as pixels, bins, or cells. In other words, these geographic portions214 may be units of the coverage area 104 _(area). For example, when thecoverage area 104 _(area) for a base station 104 is one hundred squarekilometers, the coverage area 104 _(area) is divided into a 10×10 gridwith units of one square kilometer geographic portions 214. Here, moregranular geographic portions 214 (i.e. smaller geographic areas) maycorrespond to greater accuracy at the modeler 300 (e.g., more accurateestimated characteristics 312 for a base station 104). Once the mapper200 generates at least one map 210, the at least one map 210 is then fedas an input into a model 310 of the modeler 300 to output estimatedcharacteristics 312 for the base station 104.

The modeler 300 is generally configured to receive inputs (e.g., maps210 from the mapper 200) and output estimated characteristics 312 of abase station 104 associated with the map(s) 210. The modeler 300 may bedesigned such that it may receive any number of maps 210 generated bythe mapper 200. In some examples, the modeler 300 has models 310 thatcorrespond to different combination of maps 210. For example, an inputof a particular combination of maps 210 outputs a particular estimatedcharacteristic 312 for the base station 104. Some examples of theseestimated characteristics 312 include an estimated location (e.g.,coordinate location) of the base station 104, an estimated pointingdirection of the base station 104, an antenna azimuth of the basestation 104, or an uncertainty estimate related to a characteristic ofthe base station 104 or UE device 102.

In some examples, the models 310 of the modeler 300 correspond toalgorithms configured to determine the output of a particular estimatedcharacteristic 312. In other examples, a model 310 is a machine learningmodel 310 where the model 310 is taught (or trained) based on data setsand result sets to predict its own output based on input data similar tothe data sets. For example, the model 310 receives radio signalmeasurements 132 associated with a base station 104 and based on anaggregate of the radio signal measurements 132, predicts the location ofthe base station 104 as an estimated characteristic 312. In someexamples, operators of a base station 104 train a model 310 withtraining data corresponding to the base station 104 and, based oniterative learning, enable the model 310 to determine estimatedcharacteristics 312 for a different base station 104 and/or differentoperator based on observations 130 associated with that different basestation 104. In other examples, the operators of a base station 104 whotrain a model 310 may then use the model 310 to determine estimatedcharacteristics 312. For instance, the estimated characteristics 312 canidentify anomalies when compared to expected characteristics. Detectionof these anomalies has the benefit that it may inform an operator ofpotential issues related to the base station 104 (e.g., discrepancieswith antenna azimuth, pointing direction, or location) or confirm issuesbeing experienced by UE devices 102.

Additionally or alternatively, the model 310 is a neural network 310that is fed the map(s) 210 as inputs and configured to output theestimated characteristics 312 of the base station 104. The neuralnetwork 310 may be a convolution neural network (CNN) or a deep neuralnetwork (DNN). In some examples, the model 310 is a combination of aconvolution neutral network and a deep neutral network such that theconvolution neural network filters, pools, then flattens information tosend to a deep neural network. Much like a machine learning model 310, aneural network 310 is trained to generate meaningful outputs that may beused as accurate estimated characteristics 312. For example, whentraining a neural network 310 to output the estimated characteristic 312of the location of a base station 104, a mean squared error lossfunction trains the neural network 310. Typically for training purposes,data is segregated into training and evaluation sets (e.g., 90% trainingand 10% evaluation) and the neural network 310 is trained until aperformance of the neural network 310 on the evaluation set stopsdecreasing. Once the performance stops decreasing on the evaluation set,the neural network 310 may be ready to determine estimatedcharacteristics 312 based on map(s) 210 with map information 212 relatedto observations 130.

As an illustration, the neural network 310 may function like an imagerecognition neural network. In an image recognition neural network, theneural network receives layers of an image. For example, these layersmay correspond to colors, such as red, green, and blue (i.e. RGB). Withlayers of the image and a trained familiarity with the colors red,green, and blue, the neural network is configured to identify the image.Here, much like the color layers (i.e. red, green, and blue), themodeler 300 receives map(s) 210. With the map(s) 210 and a trainedfamiliarity with the types of maps 210 provided to the modeler 300, themodel 310 (e.g., the neural network 310) is configured to identifyestimated characteristics 312 of the base station 104.

FIG. 2A is an example of a mapper 200 generating maps 210 for a basestation 104. FIG. 2A depicts the generated map 210 as a coverage map210, 210 a. The coverage map 210 a corresponds to a geographic areareferred to as the coverage area 104 _(area) of a base station 104.Typically, the mapper 200 determines the coverage area 104 _(area) ofthe coverage map 210 a from each observation 130 received by the dataprocessing hardware 122. An observation 130 commonly contains data(e.g., metadata) identifying the base station 104 emitting the radiosignal S measured as the radio signal measurement 132 of the observation130. Therefore, a collection of all observations 130, 130 a-n associatedwith a base station 104 define geographic boundaries of the coverage map210 a.

In some examples, the coverage map 210 a indicates, as map information212, at least one radio signal characteristic 212 a of the emitted radiosignal S about the base station 104. The radio signal characteristic 212a may relate to a single received observation 130 or a collection ofreceived observations 130. In some implementations, the mapper 200determines an average radio signal measurement 212 a for a cluster ofradio signal measurements 132 a in a similar location 134 a. In otherimplementations, the mapper 200 determines a variance 212 acorresponding to a cluster radio signal measurements 132 a in a similarlocation 134 a Additionally or alternatively, the mapper 200 maydetermine both an average and a variance for a cluster of radio signalmeasurements 132 a in a similar location 134 a as radio signalcharacteristics 212.

In some configurations, the mapper 200 generates the coverage map 210 afor the base station 104 by dividing a coverage area 104 a into pixels(e.g., geographic portions 214). For example, FIG. 2A illustrates thecoverage map 210 a divided into a matrix (or grid). Each cell of thismatrix (or grid) may represent a pixel referring to a geographic portion214. As seen in the example FIG. 2A, for each observation 130, 130 a-n,the mapper 200 identifies the pixel having the corresponding geographicportion 214 of the coverage area 104 _(area) that contains the location134 of the radio signal measurement 132 of the respective observation130. The mapper 200 may associate the observation 130 with theidentified pixel. For example, observation 130 a has a signalmeasurement 132 a at location 134 a that corresponds to geographicportion 214 ₃₂, where “32” is the third row and the second column pixelassociated with location 134 a Similarly, location 134 b of observation130 b corresponds to geographic portion 214 ₄₂. Location 134 c ofobservation 130 c also maps to geographic portion 214 ₄₂. Location 134 dof observation 130 d corresponds to geographic portion 214 ₄₃ andlocation 134 n of observation 130 n maps to geographic portion 214 ₄₃ aswell. With the locations 134 of each observation 130 for the basestation 104 mapped along with the radio signal measurement 132 at thatlocation 134, the mapper 200 may generate radio signal characteristics212 as map information 212 for each pixel. As previously stated, foreach pixel, the mapper 200 may determine the average (e.g., a RSRPvalue) and/or the variance of the radio signal measurements 132 of anyobservations 130 associated with the respective pixel. Referring furtherto FIG. 2A, the mapper 200 generates the average (e.g., a RSRP value)and/or the variance as radio signal characteristics 212 a ₃₂, 212 a ₄₃,and 212 a ₄₂ for the respective pixels 214 ₃₂, 214 ₄₃, and 214 ₄₂.Additionally or alternatively, the coverage map 210 a may include datastructures, referred to as bins, for each pixel, cell, and/or geographicportion 214 where an observation 130 associated with the pixel, cell,and/or geographic portion 214 may be stored. An advantage of bins isthat the data structure of the coverage map 210 a may enable to mapper200 to quickly generate additional maps 210 (e.g., observation maps 210b) from the observations 130 within the bins. This may prevent themapper 200 from once again mapping the locations 134 of the observations130.

FIG. 2B is an example of the mapper 200 generating an observation map210, 210 b based on the coverage map 210 a and the observations 130.With respect to the observation map 210 b, the map information 212refers to a metric 212 b, generated by the mapper 200, that expressesthe presence of an observation 130 in a geographic portion 214 (e.g.,pixel or cell) of the coverage map 210 a The metric 212 b may be anumber N of observations 130 associated with a geographic portion 214(e.g., pixel or cell), a log of the number N of any observations 130associated with a respective geographic portion 214 (e.g., pixel orcell), or a monotonic function of the number N of any observations 130associated with the respective geographic portion 214 (e.g., pixel orcell).

In some configurations, the mapper 200 generates the metric 212 b bydetermining a number N of any observations 130 associated with arespective geographic portion 214 (e.g., pixel or cell). When the numberN is greater than zero, the mapper 200 assigns a value to the respectivegeographic portion 214 (e.g., pixel or cell). The value may be a “1” ora number that corresponds to the number N of observations 130. When thenumber N equals zero, the mapper 200 assigns the value of “0” to therespective geographic portion 214 (e.g., pixel or cell). For exampleFIG. 2B illustrates the metric 212 b is a binary metric where a “0” isassigned to a geographic portion 214 when the geographic portion 214does not contain any observation 130 and where a “1” is assigned to ageographic portion 214 when the geographic portion 214 contains one ormore observation 130. In this example, only geographic portions 214 ₃₂,214 ₄₃, and 214 ₄₂ of the observation map 210 b ₁ are assigned a valueof “1”; while all other geographic portions 214 are assigned a value of“0.” In some examples, the modeler 300 may skip geographic portion 214assigned a value of “0.” In models 310 with machine learning and/orneutral networks, the model 310 may, over time, also learn to skipgeographic portion 214 assigned a value of “0” during the modelingprocess. In other implementations, such as FIG. 2C, the value assignedas a metric 212 b to a geographic portion 214 of the observation map 210b ₂ directly corresponds to the number N of observations 130 within thatgeographic portion 214. Here, because geographic portions 214 ₃₂, 214₄₃, and 214 ₄₂ correspond to 1, 2, and 3 observations 130, theobservation map 210 b ₂ assigns geographic portions 214 ₃₂, 214 ₄₃, and214 ₄₂ a value of 1, 2, and 3, respectively.

Although the observation map 210 b may seem like a redundant map toprovide to the modeler 300 in light of the coverage map 210 a, each map210 a, 210 b provided to the modeler 300 may provide different degreesof accuracy for outputs (e.g., the estimated characteristic 312)determined by the modeler 300. In some examples, the modeler 300 outputsa confidence indicator 314 for the estimated characteristic 312. Here,the observation map 210 b enables the modeler 300 to output a confidenceindicator 314 for the estimated characteristic 312 based on the numberof observations 130 indicated in provided observation maps 210 b Forexample, the modeler 300 has a higher confidence indicator 314 when theobservation map 210 b corresponds to a significant number ofobservations 130. In other words, the modeler 300 gains confidence inthe estimated characteristics 312 of the base station 104 when theestimated characteristics 312 are based on a larger volume ofobservations 130 (i.e., data measurements to generate the estimatedcharacteristics 312). In yet other examples, the modeler 300 outputs aconfidence indictor 314 without an observation map 210 b such that amodel 310 of the modeler 300 may be configured to determine theconfidence indicator 314 based on other map information 212 providedwith the map(s) 210 input into the modeler 300.

FIGS. 2D and 2E illustrate that other maps 210 may be provided to themodeler 300 to determine an estimated characteristic 312 for a basestation 104. In FIGS. 2D and 2E, the mapper 200 generates a terrain map210, 210 c 1.2. The terrain map 210 c may include at least one of aterrain altitude of a geographical area associated with the coveragearea 104 _(area) or a presence and/or height of objects extending abovea ground surface of the geographical area as the map information 212 c.For example, the mapper 200 generates the terrain map 210 c from terrainimages, such as topography (FIG. 2E) or images of the coverage area 104_(area) (FIG. 2D) from which the mapper 200 can determine the presenceand/or height of objects. With these images, the mapper 200 may generatevarious types of terrain maps 210 c depending on the granularity desiredby an entity such as an operator or a network provider seeking todetermine an accurate estimated characteristic 312 of the base station104. In other words, the complexity of the terrain map 210 c that themapper 200 may generate may vary depending on application and/or designof the modeler 300 and model(s) 310. For example, FIG. 2D illustrates asimple terrain map 210 c where, much like the observation map 210 b ₁,the mapper 200 generates metrics 212 c corresponding the terrain (e.g.,based on a terrain image). In FIG. 2D, the mapper 200 uses a binarymetric 212 c to assign geographic portions 214 with a “1” value when theterrain contains an object or feature (e.g., a river or a forest) and toassign geographic portions 214 with a “0” when the geographic portion214 does not contain a terrain object or feature. For example, theterrain map 210 c ₁ of FIG. 2D includes a “1” value in each geographicportion 214 (e.g., pixel or cell) of columns one and two because of theriver and the forest in the terrain image.

FIG. 2E is a more complicated terrain map 210 c 2. The terrain imagecorresponding to terrain map 210 c ₂ is a topographical map identifyingelevation within the coverage area 104 _(area). The mapper 200 maygenerate several different terrain maps 210 c with varying mapinformation 212 relating to the elevation information. Some examplesinclude, terrain maps 210 c designing, as the map information 212, theextrema within a geographic portion 214, the average elevation for therespective geographic portion 214, and/or a metric to designateelevations that may be predetermined to pose an issue with determiningestimated characteristics 312 for a base station 104. In the example ofFIG. 2E, each geographic portion 214 has map information 212 ccorresponding to the maxima elevation height within the geographicportion 214. For example, geographic portion 2145 has a maxima elevationheight of 2,000 feet based on the topography image.

In other configurations, the mapper 200 providers the modeler 300 withanother map 210, a timing advance map 210 d, whose map information 212includes a measured timing advance 212 d. The measured timing advance212 d corresponds to an estimate of how far the UE device 102 is fromthe base station 104. Generally, the timing advance is a measurementperformed by a UE device 102 such that, for example, when the UE device102 generates radio signal measurements 132 (e.g., RSSI, RSRP, or RSRQ),the UE device 102 also measures the timing advance (i.e. generates ameasured timing advance 212 d).

In other examples, the mapper 200 provides the modeler 300 with alocation uncertainty map 210 e. For example, the location uncertaintymap 210 e corresponds to a radio signal measurement 132 (e.g., signalstrength measurement such as RSSI, RSRP, RSRQ) generated by the UEdevice 102 at a given location (e.g., location X) where the radio signalmeasurement 132 also indicates a location uncertainty 212 e of somedistance Y. The location uncertainty map 210 e may be a map 210 of thelocation uncertainty 212 e or a map 210 of a statistic relating to thelocation uncertainty 212 e. For example, the location uncertainty map210 e is divided into geographic portions 214 (e.g., pixel or cell)where each geographic portion 214 represents an average of locationuncertainties 212 e from radio signal measurements 132 within ageographic portion 214. Although coverage maps 210 a, observations maps210 b, and terrain maps 210 c are discussed in detail, any map 210 maybe provided to the modeler 300 to compile more data for the model 300 toeffectively provide an estimated characteristic 312 for a base station104.

FIGS. 3A-3C are examples of the modeler 300. The modeler 300 isconfigured to receive maps 210 and/or side information 160. With themap(s) 210 and/or side information 160, the modeler 300 uses at leastone model 310 to determine an estimated characteristic 312 and/orconfidence indicator 314 for the estimated characteristic 312. The sideinformation 160 fed into the modeler 300 may include at least one offrequency of operation of a base station 104, a height of an antenna ofthe base station 104, an antenna beam width, an antenna tilt angle, apredetermined (i.e. expected) location 104 Lexp of the base station 104,or any other parameter associated with the base station 104. Inconfigurations where the modeler 300 receives side information 160 inaddition to the map(s) 210, the modeler 300 does not receive, as sideinformation 160, information related to the estimated characteristic312. The information related to the estimated characteristic 312 may belater compared to the estimated characteristic 312 for identification ofanomalies or discrepancies with the network 100. In other words, themodeler 300 may be configured such that estimated characteristic 312(i.e. output of the modeler 300) is never an input (e.g., sideinformation 160 or map information 212).

FIG. 3B is an example where the modeler 300 does not receive sideinformation 160. Here, the inputs to the modeler 300 are the coveragemap 210 a indicating radio signal characteristics 212 a and theobservation map 210 b based on metrics 212 b relating to observations130. In some examples, the modeler 300 receives more than oneobservation map 210 b as an input (e.g., a first observation map 210 b ₁with a binary metric 212 b and a second observation map 210 b ₂ with ametric 212 b relating to the number N of observations 130). Using atleast one model 310 (e.g., a neural network 310), the modeler 300outputs an estimated characteristic 312 (e.g., the estimated location312 a of the base station 104). As shown in FIGS. 3B and 3C, theestimated location 312 a of the base station 104 may be mapped and/orcompared to a map 210 identifying an expected location 104 _(Lexp) ofthe base station 104 FIGS. 3B and 3C indicate the estimated location 312a with a circle containing an “X” and the expected location 104 _(Lexp)of the base station 104 with a bullseye shape. In FIG. 3B, the expectedlocation 104 _(Lexp) of the base station 104 and the estimated location312 a are nearby and not a significant discrepancy. FIG. 3C, on theother hand, indicates that the estimated location 312 a of the basestation 104 significantly deviate from the expected location 104 _(Lexp)of the base station 104. With an identified discrepancy or anomaly, anentity (e.g., network administrator, operator or network provider) maylook to the confidence indicator 314 to further understand theidentified discrepancy or anomaly. In some implementations, the entityidentifies a ratio of the prediction error to the uncertainty (i.e.confidence indicator 314) where the prediction error corresponds to theapparent deviation between the estimated characteristic 312 and theexpected characteristic. For example, a large prediction error betweenthe expected location 104 _(Lexp) of the base station 104 and theestimated location 312 a (e.g., FIG. 3C) appears justified by a largeuncertainty for the confidence indicator 314; whereas, a largeprediction error between the expected location 104 _(Lexp) of the basestation 104 and the estimated location 312 a appears troublesome with ahigh confidence indicator 314.

FIG. 4 is another example of a communication network 400. In thisexample, the mapper 200 may receive the observations 130 from dataprocessing hardware 122 of the network gateway device 120 or from atopology data store 410 (e.g., a central server). The topology datastore 410 may continuously scan the communication network 400 to receivemeasurement data from UE devices 102. The measurements may then beinterpreted as observations 130 that the mapper 200 may use to generatemaps 210 for the model 310 to determine an estimated characteristic 312for a base station 104. With continuous scanning of the topology datastore 410, the communication network 400 may seek to resolve issues thatUE devices 102 encounter with the communication network 400 using thecombination of the mapper 200 and the modeler 300 to detect issues withbase station parameters. In some examples, an estimated characteristicstore 420 stores the estimated characteristics 312 determined by themodeler 300. By storing the estimated characteristics 312, thecommunication network 400 may be able to queue encountered issues tolater repair and resolve. In some examples, these identified issues areresolved at the base station 104 and/or UE device 102. Additionally oralternatively, the estimated characteristic store 420 may provide a datasets for models 310 of the modeler 300 to further learn via machinelearning models and/or neural network models. Learning from these datasets may further hone a predictive accuracy of the models 310 of themodeler 300 to determine estimated characteristics 312 of the basestation 104. In some configurations, the estimated characteristic store420 and/or the topology data store 410 can create numerous data setsand/or maps to utilize the data contained therein to train a model 310of the modeler 300. For example, the observations 130 sent from the UEdevices 102 and collected by the topology data store 410 may create morethan one learning map 210 from the same set of data by translating,rotating, or inverting about an axis an initial generated map 210 of aset of data. With potentially volumes of observations 130 collected atthe topology data store 410 combined with the ability to generatemultiple learning maps 210 for a model 310 with one data set, thetopology data store 410 may function as a powerful tool to train models310 for the modeler 300.

FIG. 5 illustrates a method 500 for detecting radio coverage problemsbased on UE devices 102 and observations 130 within a coverage area 104_(area) of a base station 104. While the method is described withrespect to radio base stations, the method may also be applied totelevision stations, WiFi access points and coverage maps, and othercommunication devices. At block 502, the method 500 includes receivingfrom at least one UE 102 observations 130. Each observation 130including a radio signal measurement 132 from a base station 104 and acorresponding location 134 of the radio signal S. At block 504, themethod 500 further includes generating, by data processing hardware, acoverage map 210 a for the base station 104 based on the receivedobservations 130. The coverage map 210 a indicates a radio signalcharacteristic 212 a of the emitted signal S of about the base station104. At block 506, the method 500 also includes generating anobservation map 210 b based on the coverage map 210 a and theobservations 130. At block 508, the method 500 further includesdetermining, by data processing hardware, an estimated characteristic312 of the base station 104 by feeding the coverage map 210 a and theobservation map 210 b into a neural network 310 configured to output theestimated characteristic 312 of the base station 104.

FIG. 6 is schematic view of an example computing device 600 that may beused to implement the systems and methods described in this document.The computing device 600 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

The computing device 600 includes a processor 610, memory 620, a storagedevice 630, a high-speed interface/controller 640 connecting to thememory 620 and high-speed expansion ports 650, and a low speedinterface/controller 660 connecting to a low speed bus 670 and a storagedevice 630. Each of the components 610, 620, 630, 640, 650, and 660, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 610 canprocess instructions for execution within the computing device 600,including instructions stored in the memory 620 or on the storage device630 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 680 coupled tohigh speed interface 640. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 600 maybe connected, with each device providing portions of the necessaryoperations (e.g. as a server bank, a group of blade servers, or amulti-processor system).

The memory 620 stores information non-transitorily within the computingdevice 600. The memory 620 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 620 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 600.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM)(e.g., typically used forfirmware, such as boot programs). Examples of volatile memory include,but are not limited to, random access memory (RAM), dynamic randomaccess memory (DRAM), static random access memory (SRAM), phase changememory (PCM) as well as disks or tapes.

The storage device 630 is capable of providing mass storage for thecomputing device 600. In some implementations, the storage device 630 isa computer-readable medium. In various different implementations, thestorage device 630 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 620, the storage device 630,or memory on processor 610.

The high speed controller 640 manages bandwidth-intensive operations forthe computing device 600, while the low speed controller 660 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 640is coupled to the memory 620, the display 680 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 660,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 660 is coupled to the storagedevice 630 and a low-speed expansion port 690. The low-speed expansionport 690, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 600 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 600 a or multiple times in a group of such servers 600a, as a laptop computer 600 b, or as part of a rack server system 600 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry. e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well, for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user, for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method comprising: feeding, by data processinghardware, a predicted location of a base station into a neural network,the neural network configured to output an estimated characteristic ofthe base station and to output a confidence indicator for the estimatedcharacteristic of the base station, the confidence indicatorrepresenting an uncertainty of the estimated characteristic;determining, by the data processing hardware, the estimatedcharacteristic of the base station by feeding a coverage map into theneural network, the coverage map indicating a signal characteristic ofthe emitted signal about the base station, the estimated characteristicof the base station comprising an estimated location of the basestation; determining, by the data processing hardware, a predictionerror based on (i) a deviation between the predicted location of thebase station and the estimated location of the base station and (ii) theconfidence indicator associated with the estimated characteristic outputby the neural network; and communicating, by the data processinghardware, the prediction error to an issue queue at a network entityadministrator.
 2. The method of claim 1, wherein the coverage map forthe base station is based on observations, each observation comprising:a signal measurement of a signal emitted from the base station to acorresponding user equipment (UE) device; and a location of thecorresponding UE device at a time the corresponding UE device eithermeasures the signal measurement or transmits the observation.
 3. Themethod of claim 2, wherein the coverage map for the base station isgenerated by: dividing a coverage area about the base station intopixels, each pixel corresponding to a geographical portion of thecoverage area; for each observation: identifying the pixel having thecorresponding geographical portion of the coverage area that containsthe location of the observation; and associating the observation withthe identified pixel; and for each pixel, averaging the signalmeasurements of any observations associated with a respective pixel. 4.The method of claim 2, wherein determining the estimated characteristicof the base station further comprises feeding the coverage map and anobservation map into the neural network, the observation map generatedby, for each pixel of the coverage map, generating a metric thatmonotonically expresses a number of any observations associated with therespective pixel.
 5. The method of claim 4, wherein the metric comprisesat least one of: a number of any observations associated with therespective pixel; a log of the number of any observations associatedwith the respective pixel; or a monotonic function of the number of anyobservations associated with the respective pixel.
 6. The method ofclaim 5, wherein generating the metric comprises: determining a numberof any observations associated with the respective pixel; when thenumber is greater than zero, assigning a value of the respective pixelto one; and when the number equals zero, assigning the value of therespective pixel to zero.
 7. The method of claim 2, wherein determiningthe estimated characteristic of the base station further comprisesfeeding the coverage map and a location uncertainty map into the neuralnetwork, wherein each observation comprises a location uncertaintymeasurement, and the location uncertainty map is generated based on thelocation uncertainty measurements.
 8. The method of claim 1, furthercomprising feeding a terrain map of a geographical area about the basestation into the neural network, the terrain map describing at least oneof: a terrain altitude of the geographical area; or a presence and/orheight of objects extending above a ground surface of the geographicalarea.
 9. The method of claim 1, further comprising feeding, by the dataprocessing hardware, side information into the neural network, the sideinformation comprising at least one of a frequency of operation of thebase station, a height of an antenna of the base station, an antennabeam width, or an antenna tilt angle.
 10. The method of claim 1, whereinthe estimated characteristic of the base station further comprises anestimated pointing direction of the base station or an antenna azimuthof the base station.
 11. A system comprising: data processing hardware;and memory hardware in communication with the data processing hardware,the memory hardware storing instructions that when executed on the dataprocessing hardware cause the data processing hardware to performoperations comprising: feeding a predicted location of a base stationinto a neural network, the neural network configured to output anestimated characteristic of the base station and to output a confidenceindicator for the estimated characteristic of the base station, theconfidence indicator representing an uncertainty of the estimatedcharacteristic; determining the estimated characteristic of the basestation by feeding a coverage map into the neural network, the coveragemap indicating a signal characteristic of the emitted signal about thebase station, the estimated characteristic of the base stationcomprising an estimated location of the base station; determining aprediction error based on (i) a deviation between the predicted locationof the base station and the estimated location of the base station and(ii) the confidence indicator associated with the estimatedcharacteristic output by the neural network; and communicating theprediction error to an issue queue at a network entity administrator.12. The system of claim 11, wherein the coverage map for the basestation is based on observations, each observation comprising: a signalmeasurement of a signal emitted from the base station to a correspondinguser equipment (UE) device; and a location of the corresponding UEdevice at a time the corresponding UE device either measures the signalmeasurement or transmits the observation.
 13. The system of claim 12,wherein the coverage map for the base station is generated by: dividinga coverage area about the base station into pixels, each pixelcorresponding to a geographical portion of the coverage area; for eachobservation: identifying the pixel having the corresponding geographicalportion of the coverage area that contains the location of theobservation; and associating the observation with the identified pixel;and for each pixel, averaging the signal measurements of anyobservations associated with a respective pixel.
 14. The system of claim12, wherein determining the estimated characteristic of the base stationfurther comprises feeding the coverage map and an observation map intothe neural network, the observation map generated by, for each pixel ofthe coverage map, generating a metric that monotonically expresses anumber of any observations associated with the respective pixel.
 15. Thesystem of claim 14, wherein the metric comprises at least one of: anumber of any observations associated with the respective pixel; a logof the number of any observations associated with the respective pixel;or a monotonic function of the number of any observations associatedwith the respective pixel.
 16. The system of claim 15, whereingenerating the metric comprises: determining a number of anyobservations associated with the respective pixel; when the number isgreater than zero, assigning a value of the respective pixel to one; andwhen the number equals zero, assigning the value of the respective pixelto zero.
 17. The system of claim 12, wherein determining the estimatedcharacteristic of the base station further comprises feeding thecoverage map and a location uncertainty map into the neural network,wherein each observation comprises a location uncertainty measurement,and the location uncertainty map is generated based on the locationuncertainty measurements.
 18. The system of claim 11, further comprisingfeeding a terrain map of a geographical area about the base station intothe neural network, the terrain map describing at least one of: aterrain altitude of the geographical area; or a presence and/or heightof objects extending above a ground surface of the geographical area.19. The system of claim 11, further comprising feeding, by the dataprocessing hardware, side information into the neural network, the sideinformation comprising at least one of a frequency of operation of thebase station, a height of an antenna of the base station, an antennabeam width, or an antenna tilt angle.
 20. The system of claim 11,wherein the estimated characteristic of the base station furthercomprises an estimated pointing direction of the base station or anantenna azimuth of the base station.